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ABSTRACT 

Due to the anonymous nature of online courses, many instructors require some type of proctored, comprehensive exam 
at the conclusion of the course. While comprehensive final exams are not unique to the online classroom, the integration 
of a proctored, high-stakes assessment helps to ensure that the student registered for a course is actually the student who 
is completing course assignments and activities. While this type of approach can ensure the identity of online students, it 
can be cognitively demanding for students to master the wide-range of material covered in a single, comprehensive 
exam. The purpose of the current study is to examine the impact of required weekly cumulative quizzes on students' 
retention and integration of information. One online class completed weekly assignments with optional cumulative 
review quizzes while another online class completed identical assignments plus required cumulative review quizzes. 
Results indicated that students required to complete the weekly cumulative reviews retained more information as 
indicated by significantly higher final exam scores than students with access to the optional comprehensive review 
quizzes. Discussion highlights the value of integrating mastery-learning strategies, such as comprehensive review 
quizzes, into the course mix. 

Underlying these concerns is the foct that many students 
lack the metacognitive ability to accurately judge the 
importance and significance of concepts embedded 
within a course (McKeachie, Pintrich, Lin, Smith & 
Sharman, 1990). Thus, students become overloaded with 
the daunting task of "learning everything," This overload is 
one of the main factors leading students to approach 
learning as a shallow memorization task rather than 
studying for deep understanding. 

Compounding the problem, most students dedicate their 
study time and energy to learning the information that is 
required for a specific assignment or activity (McKeachie, 
1999). Since most course activities focus on discrete 
course topics or ideas, and are not comprehensive, 
students do not invest the necessary time to integrate 
information. While students may understand the specific 
concepts, topics or themes, a lack of integration of these 
discrete pieces of information presents additional 
challenges when trying to demonstrate knowledge on a 
comprehensive exam that combines all course 
information, 

Current Study 

For the purposes of the current study, mastery as a 


INTRODUCTION 

Ensuring whether registered students are completing 
course requirements is a key challenge in achieving 
academic integrity in the online environment, Although 
numerous methods can be used to confirm a registered 
student's identity, one of the most popular approaches in 
online education is to require a substantial 
comprehensive exam with a live proctor to verify the 
student's identity. While this type of high-stakes 
comprehensive exam is often a hallmark of face-to-face 
courses as well, in the online environment, this assessment 
approach plays a dual role by reducing concerns about 
a student's identity in relationship to his/her grade. 
However, reliance on high-stakes comprehensive exams 
raises a host of new questions concerning the validity and 
effectiveness of comprehensive exams for measuring 
students'mastery of course material, 

Ideally, a comprehensive exam would demonstrate true 
mastery of course material; yet in reality students often 
score poorly on comprehensive exams (McKeachie, 
1999). Furthermore, comprehensive exam outcomes 
may relate more strongly to the mastery of test-taking 
strategies than to the mastery of course material. 
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learning outcome is defined by students' ability to 
synthesize and integrate learning across time, as 
demonstrated by success on a cumulative final 
assessment. Promoting mastery learning, then, becomes 
key to effective online course design. Toward that end, the 
purpose of this study is to examine the effectiveness of 
online cumulative review questions in promoting retention 
of key course concepts. Specifically, the study 
investigates whether online students complete mastery- 
based weekly online cumulative review assessments, 
and whether it would show increased retention of course 
material as indicated by enhanced performance on the 
high-stakes, comprehensive final exam. 

Literature Review 

Value of Assessments in Online Learning 

Research has repeatedly found that students' attention 
and focus are driven by the assessment measures, not the 
educational goals of a course (McKeachie, 1999; 
McKeachie, Pintrich, Lin, Smith & Sharman, 1990). Thus, if 
assessments focus primarily on discrete units of 
information, students will devote their time and effort 
toward the shallow memorization of these apparently 
unrelated facts and definitions. On the other hand, if 
assessments require students to demonstrate a more 
complex understanding, students will concentrate their 
effort on acquiring the relevant skills. This leaves the 
instructor with the task of implementing measures that 
accurately reflect the desired educational objectives. If 
an instructor wants learners to be able to effectively 
integrate the cumulated knowledge across a number of 
units, it is important that assessments require students to 
practice this type of integrated, cumulative 
understanding. Combining the value of mastery learning 
strategies with the necessity for practice in cumulative 
understanding, the use of cumulative mastery learning 
quizzes in an online course has the potential to enhance 
student understanding while allowing instructors to 
maintain the use of high-stakes, proctored exams to 
ensure students' identity in the faceless environment of 
online education, 

Within an online course, study questions provide a means 


by which instructors can focus students' attention and 
highlight areas of emphasis. This focus helps to decrease 
overreliance on text vocabulary and factual information 
by reducing the cognitive demands placed on the 
student. In addition, research in cognitive psychology has 
highlighted the constructive nature of learning, 
emphasizing that effective learning is the result of 
incorporating new information with prior knowledge 
(McKeachie, 1999). Thus, to promote optimal learning, 
instructors must ensure that whether the students possess 
a basic understanding from which future learning can 
build, Since many students will not initially demonstrate 
the required high level of understanding, the opportunity 
for practice and multiple reviews is vital. 

Practice, feedback and the use of multiple reviews form 
the foundation of mastery learning approaches to 
education. Mastery learning strategies are widely 
accepted as an effective and efficient method of 
instruction for post-secondary education (Guskey & 
Gates, 1985; Kulik & Kulik, 1979; Kulik, Kulik, & Bangert- 
Downs, 1090; Kulik, Kulik, & Cohen, 1979). Mastery 
learning is based on the notion that "when instruction is of 
good quality, appropriate, and adequate time is spent on 
learning, almost every student can master basic skills and 
acquire the knowledge that forms the core of a school's 
curriculum" (Montazemi & Wang, 1995, p. 186), Typically, 
mastery learning involves breaking instructional material 
into small units and requiring students to demonstrate 
mastery (a designated percentage or higher) on each 
unit before proceeding to the next, Keller's (1968) seminal 
work on mastery learning outlined a specific teaching 
methodology based on mastery learning theories. Keller's 
Personalized System of Instruction (PSI or Keller plan) has 
several components: 

• instructional material is presented via written text 

• material is divided into small units as outlined by study 
objectives 

• students proceed through the instructional units at 
their own pace 

• students must demonstrate mastery on each unit 
before moving to the next (if a student fails a unit test, 
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she/he must retest until mastery is reached) 

• individual feedback is provided immediately after 
each testing session 

By utilizing mastery learning strategies, students take 
responsibility for their own learning and have the 
opportunity to learn from their mistakes. The instructional 
tools available in an online course management system 
form an ideal environment for incorporating 
individualized, repeatable mastery learning approaches. 
Research on Effectiveness of Mastery Learning 
The use of repeated formative assessment to facilitate 
understanding is unique to mastery learning programs. If 
a student demonstrates mastery (by reaching a minimum 
designated percentage on the unit test), he/she is 
allowed to proceed to the next unit; if the student has not 
mastered the unit, feedback is provided on errors, then 
the student retests at a later time. In this sense, unit tests 
serve a dual purpose of assessment and instruction. Key 
to the instructional aspects of mastery learning instruction 
and repeatable testing is the immediate, individualized 
feedback provided after each testing session. 
Traditionally, this feedback has been provided by human 
proctors who grade unit tests and discuss the results with 
individual students, but recent advances in computer 
technology have challenged the reliance on human 
proctors to provide instructional feedback. 

Despite the potential value of human proctors, research 
suggests that the human component is not essential to 
effective mastery instruction (Buskist, Cush, DeGrandpre, 
1991; Caldwell, Bissonnettee, Klishis, Pipley, Farudi, 
Hochstetter, & Radiker, 1978; Kulik, Kulik, & Smith, 1976). An 
early component analysis of mastery learning courses 
found that the number of unit tests, the quality of 
instructional objectives and the criteria set for mastery 
performance was considerably more important than 
human proctors (Caldwell et al., 1978). In addition, 
research by Kulik, Jaksa, and Kulik (1978) indicated that 
human proctoring does not significantly increase 
students' achievement scores. Outcome measures from 
a mastery learning course conducted completely via 
computers (see Crosbie & Kelly, 1993) found that student 


achievement was high and that "... students did not 
lament for the absence of personal proctors" (p. 369). 

Due to the success of mastery learning programs in 
promoting enhanced retention, they serve as an ideal 
model for integrating mastery learning strategies into the 
online classroom. Since mastery learning programs do 
not utilize traditional lectures or live discussion sections, 
they are a cost-effective means of handling the large 
enrollments often found with the growing popularity of 
online learning. The structure of mastery learning 
programs (repeatable testing, immediate feedback, and 
minimal live intervention) make them an ideal target for 
online instruction. 

The assessment features available in most course 
management systems provide an efficient means of 
managing mastery learning approaches. Online 
assessment programs can efficiently administer, grade, 
and provide feedback on individual student tests. In 
addition, opportunities for cheating are reduced by the 
method in which tests are compiled. Using the 
randomization features available in most online 
assessment programs, each time a student accesses an 
exam, the computer randomly selects the appropriate 
number of questions from the computerized test bank. In 
this way, each exam is a unique combination of items with 
item-responses randomly ordered. Further, security 
features of online assessment programs often allow 
instructors to control the pace and sequencing of test 
progression. Specifically, when utilizing mastery learning 
methodology, it is important to ensure that students are 
testing sequentially, have demonstrated mastery on one 
unit before moving on to the next, and take time to study 
between testing attempts (to prevent the testing from 
turning into a guessing activity- this study time allows 
students to reflect on their mistakes and correct 
conceptual errors), Most importantly, online testing tools 
allow the instructor to pre-populate the assessments with 
explanatory text so that upon completion, students can 
see an item-by-item analysis and response to their work. 
Method 

Forty-four students enrolled in the accelerated, online 
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version of an advanced psychology course participated 
in the study Students self-selected into one of two sections 
(22 students per section); both sections were taught by 
the same instructor and utilized identical instructional 
resources (textbook, assignments, papers, exams, etc.). 
Both sections were required to complete weekly 
assessments (10 multiple-choice questions) covering the 
topics and readings for that week. In addition to this 
assignment, students in the control condition were given 
optional weekly mastery-based cumulative review self- 
assessments and students in the experimental section 
were required to complete a set of weekly mastery-based 
cumulative review questions (10 multiple-choice 
questions), covering ail readings up to that point in the 
course. Both weekly cumulative reviews were identical; 
the only difference between the two groups was whether 
the review was optional or required. All assignments 
(weekly readings questions and cumulative questions) 
were conducted in mastery learning format; thus students 
could repeat the assignment as many times as they 
wished to reach the desired score. Both groups were 
informed that the weekly cumulative review would 
prepare them for the high-stakes cumulative final exam. 
Results 

A comparison between-groups ANOVA on final exam 
score indicated that students required to complete the 
weekly cumulative review scored significantly higher on 
the comprehensive final exam than students who were 
provided optional cumulative review self-assessments 
[F(l, 43) = 4.719, p=.036], The average final exam score 
for students completing the weekly cumulative 
assessment was 75.09% compared to 66.33% for 
students who were not required to complete this type of 
weekly review. 

Discussion 

As hypothesized, students who were required to complete 
weekly mastery-based cumulative reviews retained more 
information and demonstrated higher performance on 
the final exam than students completing traditional 
weekly assignments with optional cumulative reviews. Not 
surprisingly, students who were not required to complete 




Figure 1. Mean Percentage Score on Final Exam 


Mean Number of Weekly Reviews Completed 



♦ Required Review 
■ Optional Review 


Figure 2. Mean Number of Weekly Reviews Completed 


the weekly cumulative reviews did not take advantage of 
the optional review assignments until the final week 
review; this "cramming" strategy did not promote 
effective retention and integration of the material. In fact, 
students who had the optional reviews completed more 
of the final cumulative reviews, performed worse on the 
final exam than students who were required to complete 
cumulative reviews each week (Figures 1 and2). 

The mandatory weekly cumulative review prompted 
students to distribute study time throughout the eight- 
weeks of the course; this continued active engagement 
with course material promoted the encoding of smaller 
segments of information, enhanced integration of 
information, and a deeper understanding of course 
material as demonstrated by achievement on the final 
exam. The continuous requirement of cumulative 
understanding via the mastery quizzes required students 
to practice integration of information and prevented 
challenges inherent when material is learned as discrete, 
unrelated components. 

It is also interesting to note that for students who were 
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required to complete the weekly cumulative reviews, the 
mean number of assignment attempts required to 
demonstrate mastery decreased throughout the term, 
Thus, as students mastered the small segments of 
information, they became more efficient at integrating 
novel course material. On average, students required 
3,17 cumulative review assignment attempts to reach 
mastery, An examination of assignment statistics 
revealed that students spent an average of 11.37 
minutes on each review assignment. For less than an hour 
of additional homework per week, the results of this study 
indicate that students can significantly increase the 
amount of information they retain through a mastery 
approach. 

The results of this study have several important 
implications for instructors of online courses: 

• Assessments should be structured in a manner that 
promotes continued student involvement with course 
material to promote ongoing integration of course 
material. 

• Students seem more responsive to the immediate 
contexts and consequences of the online mastery 
quizzes than to the delayed reality of the final exam. 

• Utilizing a mastery-learning perspective provides 
incentive and opportunity for students to effectively 
integrate key course concepts and theories. 

As previously highlighted, one of the main advantages of 
mastery-based, online assessment is the ability to provide 
immediate feedback on individual student responses. In 
general, effective feedback provides the student with two 
types of information: verification and elaboration (Kulhavy 
& Stock, 1989). Verification is the simple judgment of 
whether an answer is correct or incorrect, while 
elaboration is the informational component providing 
relevant information to direct students toward a correct 
answer and thus toward greater comprehension. Most 
researchers now share the view that successful feedback 
(feedback that facilitates the greatest gains in student 
learning) must include both verification and elaboration, 
This combination can highlight response errors, give 
correct response options, and provide information that 


both strengthens correct responses and makes them 
more memorable. In addition, educators tend to agree 
on the educational value of providing students with 
performance feedback in a timely manner. Feedback 
effectiveness research has repeatedly shown that the 
immediate delivery of feedback promotes substantial 
learning gains over delayed feedback (Chickering & 
Gamson, 1987; Kulik & Kulik, 1988). As such, when 
structuring mastery learning quizzes, it is vital that the 
online assessment program be equipped to provide both 
verification and elaboration; and, this feedback must be 
available to the student in a timely manner, 

In addition to the educational advantages of immediate, 
elaborative feedback, feedback appears to be an 
important motivational influence for students in online 
courses. The impersonal nature of online courses often 
leads to student frustration and dissatisfaction, 
Specifically, students in online courses frequently report a 
lack of available educational resources and inadequate 
opportunities to monitor their own learning/performance, 
Online mastery quizzes allow students the opportunity to 
test their knowledge, while the immediate feedback 
provides a constant resource for advancing 
understanding. The motivational aspect of feedback 
serves the dual purpose of reinforcing correct responses 
while providing guidance to prevent frustration over 
inaccurate responses. 

Research in cognitive psychology has highlighted the 
constructive nature of learning, emphasizing that 
effective learning is the result of incorporating new 
information with prior knowledge (McKeachie, Pintrich, Lin, 
Smith & Sharman, 1990). Thus, to promote optimal 
learning, instructors must ensure that students possess a 
basic understanding from which future learning can build. 
Mastery-based online quizzes provide a method of 
ensuring students' knowledge of core concepts in an 
online course. While many students will not initially 
demonstrate the required high level of understanding, the 
opportunity for practice and multiple test attempts is vital, 
By utilizing practice and multiple test attempts, students 
take responsibility for their own learning and have the 
opportunity to improve and integrate their learning 
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throughout the course. 
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